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(54) Programmable spatially light modulated microscope and miciosoopy method 

(57) A confocal optical imaging system comprises 
light source means (1 10,310.410.710). detector means 
(150.160.350.360.450.750) with at least one two- 
dimensional detector camera, and spatial light modula- 
tor means (1 20.320.420.720) such as a digital micromir- 
ror device with a first (121a) and a second group (1 21 b) 
of modulator elements, wherein the first group of modu- 
lator elements Is adapted to illuminate an object to be 
investigated according to a predetermined pattern 
sequence of illumination spots focused to conjugate 
locations (141,341.441) of the object from which detec- 
tion light is directed to the detector means for forming a 
first image Ic. and the second group of elements Is 
adapted to illuminate the object at non-conjugate loca- 
tions and/or to direct detection light from non-conjugate 
locations of the object to the detector means for forming 
a second image l^c. In an optical imaging method using 
this system, the first and second images are collected 
simultaneously or subsequently. TTie confocal image 
can then be extracted from these images by image anal- 
ysis. 
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Description 

[0001] The invention generally relates to confocal microscopes and more particularly to programmable spatially light 
modulated or programmable anray microscopes and to a microscopy method which employs freely programmable pat- 

5 terns for illuNnation and/or detection. 

[0002] Confocal microscopy based on point scanni ng systems with conjugate pairs of illumination and detection aper- 
tures is an effective tool for imaging a microscopic object to be investigated with direct optical sectioning. The discrete 
aperture spots are illuminated in the object plane of the microscope from which reflected or fluorescent light is observed 
through the conjugate detection apertures in an image plane. Commonly used confocal miaoscopes based on scan- 

10 ning systems with mechanically translated aperture disks (so-called Nipkow disks with a plurality of apertures) or with 
rotating mirrors being adapted to scan an object with a laser beam (Confocal Laser Scanning Microscopy. CSLM). 
[0003] Both scanning systems have certain limitations. The aperture disks yield particularly a restriction of the illumi- 
nation field, a degraded contrast and high intensity losses. Typically less than 3% of the disk is transmissive since the 
spacing between the pinholes must be large to maintain the confocal effect. On the other hand, the scanning mirrors of 

IS CSLM result in a low duty cycle imposed by the sequential single point data acquisition. 

[0004] The problem of intensity losses has been countered by the introduction of aperture correlation microscopes as 
described by R. Ju§kaitis. T Wilson et al. in "Nature" (vol. 383. 1996. p. 804-806) and by T Wilson, R. JuSkaitis et al. 
in "Optics Letters" (vol. 21 , 1 996. p. 1 879-1881 ). Such a miaoscope as schematically shown in Figure 9 uses for spec- 
imen illumination a multiple-point source being formed by a combination of the light source with a programmable aper- 

20 ture mask. The detection of the light reflected by the specimen is obtained by a camera through the same aperture 
mask. The aperture mask is a fast spatial light modidator formed by an array of addressable pixels or a rotating disk 
with f ixedly impressed modulation codes. 

[0005] The mask carries a pattern of uncorrelated openings and closings increasing the transmissivity of the disk to 
about 50%. Due to the correlation avoiding coding sequence used by JuSkaitis et al.. the detected image is a superpo- 
25 sition of a confocal image with a conventional image. For obtaining a final confocal image, it is necessary to detect inde- 
pendently a separate conventional image (a g. by a blank sector in a rotating disk) to be subtracted from said 
superposition. 

[0006] This additional detection of a conventnnal image is time consunvng, so that only a restricted data aquisition 
rate is available. The applicability of the aperture correlation technique is further limited due to the restricted transmis- 

30 sivity. Therefore, fluorescence measurements are only possible in exceptional cases with high fluorescence yields. A 
corresponding increase of the illumination intensity could lead to unacceptable photodamage or bleaching reactions. 
[0007] With regard to these disadvantages, an Improved spatially light modulated microscope has been described by 
M. Liang et al. in "Optics Letters" (vol. 22, 1997, p. 751-753) and in the corresponding US patent No. 5 587 832. This 
prior art microscope is schematically shown in Figure 10. A two-dimensional spatial light modulator is formed by a dig- 

35 ital micromirror device (in the following: "DMD") which r^lects the illumination light from the source (laser or white light 
source) to the probe and the detection light from the probe to a two-dimensk)nal detector. Each micromirror of the DMD 
Is individually controllable to form an illumination and detectk)n spot or not. 

[0008] The use of a DMD as a light modulator allows the direct detection of confocal images. Furthermore it is possi- 
ble to determine the minimum confocal pattern period (distance of micromirror forming illuminating spots) without oom- 

40 promising the confocality. Nevertheless, the microscope of US patent No. 5 587 832 suffers from a limited illumination 
intensity as only a part of the object reflected light can be used for imaging. Furthermore, this light used for imaging has 
an "out-of-focus" offset influencing the SNR of the confocal image in an disadvantageous manner. Finally, the prtor art 
microscope is specialized to confocal imaging without the possibility of obtaining conventional images. 
[0009] The real-time confocal microscopy or imaging, in particular in the field of imaging biological objects like cells 

45 or parts thereof, calls for further improvements with regard to sensitivity, detection speed and for an extended applica- 
bility by tiie implementation of further measurement principles. 

[001 0] It is the object of the present invention to provide an improved device and method for confocal imaging allowing 
rapkJ data acquisition, in particular with effective optical sectioning, high spatial resolution and/or high optical efficiency 
It is a particular object of the Invention to provide rapid two- or three-dimensional imaging of biological or chemical 
so materials (e. g. living cells, reaction composites etc.) and tiius information about molecular structure and function. Due 
to tiie inherent sensitivity and selectivity, molecular fluorescence is a preferred spectroscopic phenomenon to be imple- 
mented with tiie new imaging device and method. 

[0011] The above object is solved by a confocal imaging device or metiiod comprising tiie features of claim 1 or 14, 
respectively Advantageous embodiments of tiie invention are defined in the dependent claims. 
55 [001 2] The basic idea of the inventors is the operation of a confocal optical imaging system as e. g. a programmable 
spatially light modulated confocal microscope (in the following: RAM) witfi spatial light modulator means (in the follow- 
ing: SLM) such that tiie entire light output from a specimen or object is dissected into two images being collected simul- 
taneously or subsequently. Generally, spatial light modulator means comprise an array of modulator elements the 
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transmission or reflections properties of which being individually controllable. With a first group of SLM nmlulator ele- 
ments ("on"-etements), the object is illuminated and a focal conjugate image is collected, while with a second group of 
SLM modulator elements ("off"-elements), a non-conjugate image is collected. The non-conjugate image contains out- 
of-fbcus light. The illumination spots formed by the SLM modulator elements are focused to a focal plane of the object. 

5 [0013] When positioned at the image plane of a microscope, the SLM elements each of which being conjugate to a 
distinct point in the focal plane of the object define a programmable an^ay which is used for illumination and/or detection. 
[0014] The modulator elements of the first group are individually controllable such that the pattern sequence of illu- 
mination spots is represented by a time<iependent systematically shifting grid pattern or a pattern based on a pseudo- 
random sequence of finite length. In the first case, the first image is an image corresponding to a confocal image and 

10 the secortd image is a difference image between a non-confocal image and the first image. In the second case, the first 
image is a superposition of confocal image and a non-confbcat image and the second image is a difference image 
between a non-confocal image and a confocal image. In any case, the first Image can contain a portion of a conjugate 
image as shown in Figure 2. 

[001 5] The SLM can be operated in transmission or reflection mode. Depending on the number of detector systenfis 
IS of the detection means, the PAM can be built as a so-called single path or double path PAM. According to a preferred 
arrangement, the light source means of the confocal optical imaging system contain a white light lamp and means for 
wavelength selection. 

[001 6] Preferred applications of the confocal optical imaging system are investigations In the cell and tissue biology, 
analytical/biotechnological procedures, in-situ hybridization in genetic analysis, fbrnfiation of an optical mask for position 
20 selective photochemical reactions, generation and readout of biochemical arrays on chips, targe scale surface inspec- 
tions in the semiconductor industry, and/or optical recording and read-out. 

[0017] The invention has the following advantages. The PAM can be implemented as a modular extension of existing 
microscopes. The out-of-fbcus tight normally rejected with conventional CLSM is collected and used to enhance the in- 
focus image. Each element of the first group of modulator elements individually is controllable or programmable such 
25 that the pattern sequence of illumination spots illuminates at least one predetermined region-of interest of the object. 
The implementation of a highly flexible white light microscope capable of full field and/or confocal imaging (with laser 
as option) Is available. Due to a high optical throughput, a white light illumination is allowed yielding an extended appli* 
cability of the microscope. 

[0018] The simultaneous use of botii infiages delivered via ttie "on"-elements and the "off-elements is a unique fea- 
30 ture of tiie invention allowing in particular improved deconvolution algorithms. 

[0019] Further details and advantages of the invention are described in the following with reference to tiie attached 
drawings which show in: 

Fig. 1 a schematic view of a fluorescence PAM according to the invention: 

35 

Fig. 2 a simulation of the imaging of an infinite plane in dependence on Illumination spot lattk;e distances; 
Fig. 3 a schematic view of a common path dual reflection PAM according to the invention: 
40 Fig. 4 a schematic view of a dual path single reflection PAM according to the Invention: 
Fig. 5 an illustration of the axial response of a PAM; 
Fig. 6 an illusti^ation of the sectioning and background suppression: 

45 

Figs. 7A, 7B: schematic views of a single camera common patti dual reflection PAM according to the invention; 

Fig. 8: a miaoscopic image taken according to the invention showing optical sectioning capability: 

so Fig. 9: a schematic view of an existing system for tiie generation of uncorrelated apertures using rotating disk tech- 
nology (prior art); and 

Fig. 10: a schematic view of a spatially light modulated confocal microscope (prior art). 

55 [0020] The programmable array mk;ro6Cope (PAM) according to the invention can be irrplemented on the basis of 
spatial light modulators (in the following: SLM) being operated in transmission (e. g. programmable aperture masks on 
the basis of liquid crystals or micromechanteal switches) or in reflection (e. g. OMD). In tiie following description, refer- 
ence is made in a non-restricting manner to reflection SLM with DMD (as an example). The invention can be put into 
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practice with transmission SLM in an analogous way. 

[0021] Figure 1 represents a schematic view of a fluorescence PAM 100 according to the invention. The PAM 100 
basically Includes a light source 110 which is a white lamp or a laser source, a DMD 120 operating as a SLM, imaging 
optics 130, a probe portion 140 and a detection system 150, 160. 

[0022] The DMD 120 consists of an array of square mirrors 121a, 121b,... (only one element is illustrated, resp.) each 
of which can be separately tilted into two stable resting states over a predetermined tilt angle ± a to the normal. The 
value of the tilt angle a depends on the DMD practically used. Each mirror element has a predetermined area which 
again depends on the DMD. Preferred parameters are e. g. a = ± 10* with < 20 ^m square min-ors. Each mirror element 
can be switched between the resting states with characteristic frequencies in the kHz-range according to a predeter- 
mined nxxlulatlon pattern. The resting states are called "on"- and '*off "-positions according to which the respective con- 
jugate contributions and the non-conjugate contributions of the detected light (fluorescence or reflection) are deflected 
to the detection system 150, 160. Both contributions yield a focal conjugate image 1^ and a non-conjugate image Inc. 
resp., on the basis of which the desired oonfbcal image is obtained as outlined below. 

[0023] The detection system 150, 160 is a common path dual reflection system. This nomenclature is used for the 
following reason. The "common path" refers to the fact that illumination and detection follow an essentially common 
optical path between the illumination and detection apertures. The "dual reflection" refers to the fact that the SLM ele- 
ments can direct light along two subsequent optical paths. Accordingly, the images l^c and Ic (see below) can be col- 
lected simultaneously. An alternative is described below as a dual path single reflection system (Figure 4). 
[0024] Referring again to Figure 1 . each path contains a two-dimensional camera 1 51 , 1 61 and detection optics 111, 
1 1 2, 1 62. resp. . The camera 1 51 is adapted to detect the light reflected by the mirrors in the "on"-po6ition while the canf^ 
era 161 receives light reflected in the "ofT-position. 

[0025] In the following, the detection of the focal conjugate image 1^ with camera 151 and the non-conjugate image 
Inc with camera 161 of the system according to Figure 1 is described. After the detection, further steps of image 
processing, storing, displaying etc. are added. These steps are not desaibed in detail as far as they are known from 
scanning systems. 

[0026] During the frame integration time, the modulation pattern of the SLM (DMD 1 20) is changed N times in order 
to generate a confbcal image (scanning). The i-th modulation of the SLM is denoted with: 



where x^.yd are continuous coordinates of the SLM and the shape and finite size of the elements (minrors) is included 
in Sj. 

Detection Fo cal Coniuoate Inraqe 

[0027] For obtaining the focal conjugate Image 1^, the modulations Sj are preferably selected according to a grid pat- 
tern approach or a pseudorandom sequence approach which are described in the following. The grid pattern approach 
has the advantage of easy inplementation and yielding a confocal image without post-processing. Although some post- 
processing is needed in the pseudorandom sequence approach, the frame integration time for a given signal level can 
be essentially shorter. 

(i) GrkJ Pattern Approach 

[0028] In the grid pattern approach, the elements (e. g. micromin'ors or liquid aystal elements) of the SLM are 
switched on according to a grid pattern and shifted systematically. The modulation pattern is selected according to 




1 if (x^,y^) is on an "on" element 
0 if (x^j^y^) is on an "off" element 



(1) 



Sa.b(Xd.yd) = G(^d-an.yd-bn) 



(2) 



wherein ti is the size of an element (assuming square elements with 100% fill factor) and the index i in eq. (1) has been 
replaced by two-dimensional integer indices a, b with 0^ a < nx* 0 ^ b < ny. The grid is defined by the property 
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G(Xd,yd) = G(Xd'p8^,y^-q5y) p,q gN (3) 



5 X n ^Ti and 5 y s n yti are the lattice distances of the grid. The real lattice parameters being contained in the function 
G can yield different suitable shapes. These shapes comprise e.g. square shapes, but pseudo-hexagonal grids or line 
patterns or more complex shapes are also possible. 

[0029] The focal conjugate image of an object O(xo,yo.zo) in a coordinate system (xo,yo,zo) in an object space, with a 
scanning in the axial direction by displacement over the distance Zs is given by: 

+00 ^ X 4> 4> 

lc.s(''d.y<l.Zd) 'rirllj """(^•"•S"'''*^ lQ(^-U.^-V,W)0(U.V.W-Z3)dUClvdW (4) 



wherein M is the magnification. T the total frame integration time and H^m (xo>yo>zo) the emission PSF (point spread 
function) of the objective. Iq is the total illumination according to: 

+00 

Iq = J J G(Mu.Mv)H3^(Xo-u.yo-v,Zo)dudv (5) 



wherein Hex(xo,yo.zo) the excitation PSF of the objective. The total Illumination enters eq. (4) with spatially varying shifts 
in the lateral direction given by 

<t>x = XdmodTi Oys=y^modii (6) 

[0030] The image according to eq. (4) can be viewed as a three-dimensional convolution with a spatially varying PSF, 
due to the dependence of 4>x, <J>y x^j and y^. The SLM cannot scan continuously and GCx^^.y^) can (contrary to the con- 
tinuous motion in known miaoscopes as e.g. in so-called tandem scanning microscopes) only shift over integer multi- 
ples of 11. The distinction is negligible if q/M is small compared to the sampling density of the image. For very large % a 
pixelation of the image can occur. The PSF can be made spatially invariant by scanning the SLM over n (so-called dith- 
ering). 

[0031] A numerical simulation of the response of an infinitely thin fluorescent plane accorcfing to eq. (4) is shown in 
Figure 2 as a function of the plane position relative to the focus plane. 

[0032] Tlie simulation is based on square grids of varying lattice distances with for instance the parameters 
63j = 6y = {2, 3, 5, 10,20} • ii . element size ti = 17 |im, NA= 1.4, M = 100, excitation wavelength A^x = 633 nm, emis- 
sion wavelength = 665 nm, refractive index 1.515. These parameters are examples only being adapted to a practi- 
cal implementation of the invention. The results are normalized compared with the image of a conventional microscope 
and an ideal confocal microscope. The conventional response (very lop line in Figure 2 or In the measured data of Fig- 
ure 5, see below) deviates from the expected straight line as a result of simulating an infinitely large object in a finite 
sized image. Even for relatively small lattice distances, a sectioning effect is apparent, although with an increased back- 
ground. With increased lattice distance spacing, the response becomes sharper and the background suppression is 
more efficient. A similar t>ehaviour of the offset rise as shown in Figure 2 is given when the spacings of the tiles in the 
finite length pseudo random sequences are varied. 

(ii) Pseudorandom Sequence Approach 

[0033] In the pseudorandom sequence approach, the elements of the SLM also are switched according to a pseudo- 
random sequence of finite length. The ideal situation of completely uncorrelated modulation is difficult to achieve in 
practice, since the number of times that the SLM can change its pattern within a given integration frame is limited. 
Therefore tiie complete plane Is modulated with a repeated sequence of N two-dimensional patterns Rj(a.b). The 
number N and the period of one pattern can be selected such tiiat the sequence is short compared witii the integration 
time of one frame. However, another selection is possible. The pattern sequence Rj(a.b) has the property: 
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Y Ri (a,b)Ri (c,d) = N6(a-c,b-d) a,b,c,d 6 N (7) 

i-O 



wherein Rj(a,b) has the value 1 or -1 . A pseudorandom sequence of two-dimensional patterns with the property (7) can 
be derived from an appropriate mapping of one-dimensional rows or columns of cyclic Hadamard matrices as 
described by Harwit et al. in "Hadamard Transform Optics" (Academic Press, 1979, New York). Since SjCx^.y^) can only 
10 assume values of 0 and 1 , the sequence (1 + Ri(a,b))y2 is used. Another aperture congelation sequence approach 
according to the above publication of JuSkaitis et al. can be used. 

[0034] On the basis of a grkj aooording to eq. (3) for "tiling" the i-th pattern, the conplete modulation pattern Sj(Xd.yd) 
is formed. All SLM elements for which G(x^j-an,y j,-bn) = 1 are switched in an identical manner using the sequence (1 
-I- Rj(a.b))/2. The i-th nrxxlulalion is given by a summation over all a and b as ail elements are switched simultaneously: 

IS 

S|(Xd.yd)-I £ |(1+R,(a,b))G(Xa-ati.yd-bn) (8) 

^ ^•'^E I Ri(«'WG(Xd-ati.yd-bn) 

a^O bsO 

[0035] The resulting modulation and accordingly the resulting signal consists of two parts, wherein the first part is a 
"constant" modulation yielding a conventional full light image. 
25 [0036] The focal conjugate image of the object O(xo.yo.zo) in the coordinate system (xo,yo>Zo) is given by: 

+00 

U.R(Xd.yd.28) = I IJ I "em(^-U.^-V.W) X ©(U.V.W-Z^jdudvdw + 

1 1! J "«'"(J|-u.5|-v.w) X l3(!^.u.?^-v.w)0(u.v.w.2 Jdudvdw 



35 [0037] The pseudorandom sequence approach Is very light efficient as 50% of alt SLM elements are "on". This is sup- 
ported by eq. (9) which equals to eq. (4) except of coefficient T/4 Instead of Vnj^ in the second term. Atthough both 
approaches yield the same confocal image for the same grid G ()^.yd). the pseudorandom sequence generates a much 
larger signal, whose strength Is independent of the sequence lengtfi. 

[0038] A numerical simulation of the response of an infinitely thin fluorescent plane according to eq. (9) is identical to 
40 that of Figure 2 except for a constant offset. 

[0039] To obtain the confocal image with the pseudorandom sequence approach, a compensation term must be sub- 
tracted. This may be a conventional image according to the above technique described by JuSkaitis et al. or preferably 
the non-conjugate image. In the latter case, twice the confocal signal is obtained relative to the former. 

45 Detection Non-coniuoate Image 

[0040] The non-conjugate image Inc can be imaged on the second camera 161 . Alternatively, it Is possible to image 
the non-conjugate image Inc on the same camera like the conjugate Image Ic- An example of such a single camera sys- 
tem of the common path type is illustrated below with reference to Figures 7A and 7B. According to the invention, all of 
50 the fluorescent or reflected light from the object falling on the SLM is collected by the cameras. Therefore, the sum of 
the detected images must be a conventional image, so that the non-conjugate image Inc is the difference between the 
conventional image and the confocal image given by eq. (4) or (9) . For the pseudorandom sequence, the difference 
between 1^ and i^c equals the desired confocal image. 

55 DeconvQlution 

[0041 ] If the images are obtained according to the above approaches a deconvolution algorithm facultatively can be 
implemented for a further improved reconstructk3n or restoratton of the object. This represents an essential advantage 
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as the performance of deconvoiution algorithms is strongly limited by the nature and SNR of the input image(s). While 
the detected signal of a confocal microscope can be very noisy with a degraded restoration result, the detected signal 
of a conventional microscope (which however has inferior resolution and lacks sectioning capability) is much stronger, 
allowing an improved deconvoiution. The invention allows the combination of the advantages of both the confocal and 

5 conventional imaging with deconvoiution. The above limitation is reduced according to the deconvoiution algorithm, 
wherein both images of one or more optical sections are combined in an appropriate enhancement algorithm. 
[0042] The enhancement algorithm can be a nearest-neight)or approach where three or more sections are used to 
generate one single enhanced image or an algorithm where 3D-stacks of images are used for a full 3D reconstruction 
as it is described by P. J. Verveer and T M. Jovin in "Journal of the Optical Society of America A" (vol. 14, 1997, p. 1 696- 

10 1 706) completely being included Into the present disclosure by reference. The latter model is cited in the following in 
matrix notation. 

[0043] A multi-dimensional discrete Image is represented with a vector by stacking its pixels. A multi-dimensional lin- 
ear blurring operation is represented by a matrix. This leads to the imaging equation: 

15 

1 = N [Hf+b] (10) 



where I. f and b represent the detected image, the object and a known background, respi. The blurring introduced by 
20 the microscope optics is assumed to be linear and is given by the multiplication with the matrix H. The function N[.] rep- 
resents the application of a noise process to each element of its vector argument, usually a Pbisson distribution in flu- 
orescence imaging. 

[0044] The con'esponding equations for the conjugate and non-conjugate images 
and 

can be combined according to the invention in a single equation 

35 




(11) 



40 

by stacking 1^ and Inc into a single vector and He and Hnc into a composite operator. Since equ. (11) has the same form 
as equ. (10), the available algorithm can easily modified for (1 1). 

[0045] In equ. (10) and (1 1), the dimensionality of the image is irrelevant. The implementation of H depends on the 
dimensionality, but the algorithms themselves do not. Normally, the dimensionality is equal to 3 (spatial dimensions). 
45 However, further dimensions such as spectral or time dimensions can be added with only a change of H. For instance, 
if a time dependency is measured for each image pixel or for groups of image pixels, the dimensionality is equal to 4. If 
blun-ing is present in the time dimension, it can be deoonvoluted from the data by extending H with the time dimension. 
The capability to apply these deconvoiution algorithms also in the spectral and time dimensions, whereas previoi^ly 
they have been restricted to the spatial dimensions, represents an essential advantage of the invention. 

50 

Further PAMQrr^nqOTent? 

[0046] Figure 3 shows a schematic view of a common path dual reflection RAM 300 according to the invention. As in 
Figure 1 . RAM 300 basically includes a light source 310. a SLM 320. imaging optics 330. a probe portton 340 and detec- 
55 tion systems 350. 360. RAM 300 can be implemented with a Digital Light Processing kit (e. g. of Texas Instruments. Dal- 
las. USA) as a light source in combination with a microscope with side illumination ports (e. g. Zeiss Axioplan) and CCD 
cameras (e. g. CH220 cameras of Photometries, Tucson, USA. with Kodak KAF1 400 CCD sensors). The probe portion 
340 contains a driving device for z-displacements (perpendicular to the focus plane) like a computerized focus control 
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e. g. of LudI Electronics Products, Hawthorne, USA. PAM 300 additionally contains a driving unit for controlling the mod- 
ulator elements, a control unit, calculation and deconvolution circuits and display units (not shown). The illumination 
path is hatched. 

[0047] The SLM 320 being a DMD (as desaibed above) is illuminated with the light source 310 which is a filtered 
5 white lanp. The filter can be a band pass for a predetermined wavelength range of interest. The reflections from the 
"on"-elements 321a (only one element is illustrated) are focused with the objective 342 into the object plane 341 . Fluo- 
rescence light stimulated in the object plane 341 returns through the same optical path to the SLM 320 where it is 
reflected over the semitransmissive mirror 311 along the 1^ path to the camera 351 through a filter 353 and a lens 352. 
Light originating from the out-of-focus positions (e.g. plane nc in the inset of Figure 3) is reflected by the "off"-elements 
10 321b (only one element is illustrated) over the mirror 364 along the 1^^ path through a filter 363 and a lens 362 to the 
camera 361 . The confocal arrangement allows to reject light from the focal plane as the corresponding SLM elements 
are turned off. The filters 353 and 363 are preferably long pass filters being adapted for fluorescence measurements 

(^m > 

[0048] As in Rgure 1 , the focal conjugate image 1^ is collected with camera 351 and the non-conjugate image Inc is 

75 collected with camera 361 . 

[0049] Figure 4 shows a schematic view of a dual path single reflection PAM 400 being another embodiment of the 
invention. The "dual path" refers to the fact that illumination and detection follow two different optical paths. Further, 
although the SLM is capable of directing light in two directions, only a single reflection direction is used. While in the 
embodiment of Figure 3, the confbcality is defined by the same SLM element, according to Figure 4 the confbcality is 

20 def inded by the SLM element on the illuminations side and by the CCD pixels on the detection side. 

[0050] PAM 400 is adapted in a simplified manner to a conventional miaosoope with one optical axis on which the 
probe portion 440» imaging optics 430 and the detection system 450 are arranged. The microscope can comprise the 
same basic components as noted stave. The illumination light from the light source 410 is reflected via the SLM 420 
and a side port of the microscope to a beam splitter (e. g. half-silvered mirror) in the optical axis. 

25 [0051 ] As the detection system 450 contains only one camera 451 . the separation of the Iq and Inc images is done on 
the basis of an analysis of the camera images. The dual path PAM in combination with the analysis reconstruction has 
the advantages of easy implementabillty and high efficiency and light throughput 

[0052] According to the shifting grid approach, the image pixels of the ''on''-elements are found in the recorded set by 
selecting the maximal gray values. Since each SLM element of the DMD is to be switched "on" exactly once in one 
30 series, the pixels of the "on'-states have maximum intensity. The Ic image is reconstructed from these selected pixels 
while the Inc image results from the sum of the remaining pixels. 

[0053] In the pseudorandom sequence approach, the sum of the brightest half of the pixels is the Ic image while the 
other half represents the 1^^ image. As the Ic and Inc images represent the sum and difference of a conventional and a 
confocal image, resp., the confocal image can be calculated by the difference Ic - Inc- 
35 [0054] Figure 5 shows the axial response of a PAM using a DMD with a 32 x 32 square lattice (con^esponding to 544 
^m spot distance), a 100 x 1.3NA (Numerical Aperture) oil immersion objective and X « 450 - 490 nm. The axial z-scan 
of a reflective surface yields confocal images with intensities showing a strong optical sectioning ability in accordance 
with the simulation in Rgure 2. 

[0055] An experimental verification of the supression of background signal during optical sectioning is shown in Figure 
40 6. A reflective surface (mirror) is scanned along ttie z-axis tiirough tiie focus plane. The parameters are square lattices 
with distances 6^ = 6y = (3, 5. 10, 20} • ti , ii = 17 nm (corresponding to 16 jim size plus 1 jim between mirror gap), 
a 100 X 1 .3 NA oil immersion objective and X = 450 - 490 nm. 

[0056] Figures 7 A and 7B show an alternative example of common patii dual reflection PAM 700 with two illumination 
axes using only a single camera for image collection. This concept has certain advantages as to the cost saving use of 
45 only one camera and tiie simpler optical alignment of ttie elements forming the optical paths for ttie images l^ and l^c- 
[0057] PAM 700 comprises basically similiar components as PAM 300 shown in Figure 3. in particular light sources 
710a. 710b, a SLM 720. Imaging optics 730. an objective 743. a probe portion 740 and a detection system 750. The 
collection of one full image is accomplished in two steps. For the collection of image sequences, the images Ic and ipc 
are collected alternately. 

so [0058] First ttie image Ic is collected by illuminating ttie object from a first side with the light source 710a via a semi- 
transmissive mirror 71 1 , the SLM 720 and the imaging optics 730 and detecting ttie focal plane image along the 
reversed optical path via the SiM 720, the semitransmissive mirror 71 1 . a filter 753, a lens 752 with the camera 751 . 
This situation is shown in Figure 7A. The pattern used for controlling the SLM 720 is called the "positive" representation 
corresponding to ttie pseudo-random or ottier sequences displayed on ttie SLM. 

55 [0059] Second, as shown in Figure 7B. the image Ipc is collected by illuminating ttie object from a second side (light 
source 710b) while controlling ttie SLM 720 wrtti a so-called "negative" representation where all off-elements are con- 
trolled according to the sequence used for controlling ttie on-elements in the above first step As ttie illumination light is 
directed along a second axis, being symmetrically arranged to the first step illumination the positions of ttie images Iq 
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and Inc are reversed. 

[0060] The two-sided illumination can be implemented according to various ways, e. g. t^y a single light source being 
directed by minrors along the two illumination axes or by two identical light sources. 

5 Fgrth^r advantag e? and applications 

[0061] The improved performance of a PAM according to the invention compared to a conventional Confocal Laser 
Scanning Microscope (CLSM) can be derived from the following relative parameters (PAM/CLSM): a relative irradiance 
at the focal plane, p relative detector quantum efficiency, y relative dwell time per pixel, and 5 relative number of scan- 

10 ning apertures. For the following examples, an image field of 10^ x 10^ elements (total pixel number: N = 10^) and a 
grid period n a 10 are assumed. Accordingly, 5 equals Wr? = 10^. Typical values for a and p are 10'^ (full-field lamp 
illumination compared to a diffraction limited laser source) and 10 (CCD sensor compared with photomultiplier cath* 
ode). With a dwelt time of 1 ms (PAM) and 10 ^s (CLSM), y is 100 representing a proportional increase in speed (6/y). 
The relative signal strength being described with a-p*Y would be lO-fold higher for the PAM. The pseudorandom 

IS sequence approach yields a further increase In signal level (n^/2 in this example) due to the 50 % active state of each 
SLM element. Accordingly, the pseudorandom sequence approach is preferred for microscopic investigation of biolog- 
ical samples in vivo. 

[0062] The PAM has the potential to increase the acquisition speed two orders of magnitude. Although the lamp 
source of the PAM yields a much lower irradiance at the focal plane than a laser in a CLSM. this is compensated by the 
20 high number of points that are scanned in parallel (allowing longer dwell times) and the higher quantum efficiency of the 
detector. 

[0063] Preferred applications of the invention are in the ceil and tissue biology, delivering image data as intermediate 
information for real-time medical diagnostic procedures, analytical/biotechndogical procedures (such as e. g. in-situ 
hybridization in genetic analysis), readout of biochemical arrays on chips, the large scale surface inspection e. g. In the 

25 semiconductor industry, and/or optical recording and read-out. 

[0064] A microscopic system according to the invention allows the simultaneous initiation and monitoring of photo- 
chemical reactions e. g. for protein synthesis. The initiatfon is achieved by the irradiation of predetermined substances 
with a suitable wavelength while the monitoring comprises microscopic measurements with these substances or the 
reaction products. A particular applicatfon is the formation of an optical mask for position selective photochemical reac- 

30 tions (e. g. for the synthesis of DNA sequences) on a substrate. 

Claims 

1 . Confocal optical imaging system (100. 300, 400, 700) comprising: 

35 

- light source means (110. 310. 410. 710); 

- detector means (1 50. 1 60. 350. 360, 450, 750) with at least one two-dimensional detector camera: and 

- spatial light modulator means (120. 320, 420. 720) with a first (121a) and a second group (121b) of modulator 
elements, wherein the first group of modulator elements is adapted to illuminate an object to be investigated 

40 according to a predetermined pattern sequence of illumination spots focused to conjugate locations (1 41 , 341 , 

441) of the object wherein detection light from the conjugate locations forms a first image Ic at the detector 
means. 

characterized In that 

the second group of elements is adapted to collect light at non-conjugate locations of the object wherein detec- 
ts tion light from the non-conjugate locations forms a second image Inc at the detector means. 

2. Confocal optical imaging system according to claim 1. wherein each element of the first group of modulator ele- 
ments individually is controllable such that the pattern sequence of illumination spots is represented by a time- 
dependent systematically shifting grid pattern and the first image Ic is a confocal image and the second image Inc 

so is a difference image between a non-confocal image and the first image t^. 

3. Confocal optical imaging system according to claim 1 , wherein each element of the first group of nruxiulator ele- 
ments individually is controliable such that the pattern sequence of illumination spots is represented by a time- 
dependent pattern based on a pseudo-random sequence of finite length and the first Image Ic is a superposition of 

55 a confocal image and a non-confocal image and the second image Inc is a difference image between a confocal 
image and a non-confocal image. 

4. Confocal optical imaging system according to daim 2 or 3. wherein the pattern sequence is selected to be 
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- a systematically shifting line or lines either spaced regularly, according to a pseudo-random sequence of finite 
length, according to a S-matrix type Hadamard sequence or randomly, 

- regular dot lattices as e. g. triangles, squares, rectangles, hexagons, 

- random patterns or pseudo-random patterns of finite length based on so-called Walsch, Sylvester, Hadamard 
5 or Golay sequences, 

- square or rectangular grids formed from intersecting line patterns, 

- full field "on" pattern, 

- plane filling patterns k>eing adapted to turn on the SLM elements an integral number of times when sequentially 
produced on the SLM, 

10 ' a repeated sequence of two-dimensional patterns being derived from rows or columns of cyclic Hadamard 
matrices, or 

- a combination of the above pattern sequences. 

5. Confbcal optical imaging system according to one of the foregoing claims, wherein the detector means are adapted 
IS to collect the light available at the image plane of the system by a combined detection of both images and l^c- 

6. Confocal optical imaging system according to one of the foregoing claims, wherein the spatial light modulator 
means comprise a transmitting mask or a reflecting mask. 

20 7. Confocal optical imaging system according to daim 6, wherein the reflecting mask is a digital miaomirror device 
comprising a plurality of micronfiirrors each of which can be separately tilted into two stable resting states over a 
predetermined tilt angle ± a to the normal, wherein the first and second resting states contribute to the first and sec- 
ond group of modulations elements, respectively. 

25 8. Confocal optical imaging system according to one of the foregoing claims, wherein the detector means comprise 
two detector systems each containing a two-dimensional camera for collecting the first or the second images Ic and 

tnc resp.. 

9. Confbcal optical imaging system according to one of the claims 1 to 7, wherein the detector means comprise one 
30 detector system with one two-dimensional camera for collecting the first and second images Ic and Inc simultane- 
ously or sequentially. 

10. Confocal optical Imaging system according to claim 9, wherein the camera being adapted to collect the first and 
second images simultaneously, said imaging system further comprising a circuit for image analysis being adapted 

35 to separate the first and second images Ic and Inc- 

11. Confocal optical imaging system according to claim 9, wherein the camera being adapted to collect the first and 
second images sequentielly such, that firstly each element of the first group of modulator elements individually 
t>eing controllable according to said pattern sequence for collecting the first image Iq and subsequently each ele- 

40 ment of the second group of modulator elements individually being controllable according to said pattern sequence 
for collecting the second image Inc. wherein the light source means comprise first and second light sources illumi- 
nating the object along two axes. 

12. Confocal optical imaging system according to one of the foregoing claims, wherein the detectbn light is fluores- 
45 cence or phosphorescence light or Raman-scattered light emitted from the object. 

13. Confbcal optical imaging system according to one of the foregoing claims being a part of a programmable spatially 
light modulated confbcal microscope. 

so 14. Confbcal optical imaging method, comprising the steps of: 

- directing light from light source means (110, 310, 410) to a spatial light modulator means (120. 320. 420) with 
a first (121a) and a second (121b) group of modulator elements; 

• directing light from the first group of modulator elements to an object (1 40, 340, 440) to be investigated accord- 
55 ing to a predetermined pattern sequence of illumination spots focused to conjugate locations of the object; and 

- collecting a first image Ic with detector means (150. 160. 350. 360. 450) by directing detection light from the 
corrugate locations to the detector means; 

characterized by 
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collecting detection light from non-conjugate locations of the object with the second group of modulator ele- 
ments and forming a second image Inc by directing said detection light from the non-conjugate locations to the 
detector means. 

15. Confocal optical imaging method according to daim 15, wherein each element of the first group of modulator ele- 
ments individually is controlled such that the pattern sequence of illumination spots is a dime-dependent system- 
atically shifting grid pattern illuminating at least one predetermined region-of interest of the object, and the first 
image \q is a confocal image and the second image Inc is a difference image between a non-confocal image and 

the first image Ic- 

16. Confocal optical imaging method according to daim 15. wherein each element of the first group of modulator ele- 
ments individually is controlled such that the pattern sequence of illumination spots is a time-dependent pattern 
based on a pseudo-random sequence of finite length illuminating at least one predetermined region-of interest of 
the object, and the first image 1^ is a superposition of confocal Image and a non-oonfbcal image and the second 
image Inc is a difference image between a confocal image and a non-confbcal image. 

17. Confocal optical imaging method according to daim 16, wherein each element of the first group of modulator ele- 
ments individually is controlled such that the pattern sequence of illumination spots is represented by a repeated 
sequence of two-dimensional patterns. 

18. Confocal optical imaging method according to one of the claims 14 to 17. wherein the detector means comprise 
two detector systems each containing a two-dimensional camera and the first and second images are collected by 
the two cameras, respectively. 

19. Confocal optical imaging method according to one of the daims 14 to 17, wherein the detector means comprise 
one detector system with a two-dimensional camera and the first and second images both are collected by the 
camera. 

20. Confocal optical imaging method according to daim 19, further comprising an image analysis for separating the 
first and second images, wherein in the collected image the image pixels of the first image are found by selecting 
the maximal gray values or by selecting the brightest half of the pixels. 

21. Confocal optical imaging method according to one of the daims 14 to 20, wherein the step of directing light from 
the light source means to the spatial light modulator means includes a wavelength selection. 

22. Confocal optical imaging method according to one of the claims 14 to 21 , further comprising a deconvolution pro- 
cedure wherein the object is reconstructed with a deconvolution algorithm combining both first and second images 
of one or more collected optical sections. 

23. Confocal optical imaging method according to claim 22, wherein the deconvolution algorithm includes an enhance- 
ment algorithm with a nearest-neighbor approach wherein three or more sections are used to generate one single 
enhanced image. 

24. Use of a confocal optical imaging system or method according to one of the foregoing daims for the applications: 

- investigations in cell and tissue biology, 

- simultaneous position selective initiation and monitoring photochemical reactions on a substrate, wherein the 
initiation indudes the in-adiation of predetermined substances with a suitable wavelength and/or the monitoring 
includes fluorescence measurements with the substances or the reaction products, 
analytical/biotechnological procedures, 

in-situ hybridization in genetic analysis, 

- formation of an optical mask for position selective photochemical reactions. 

position selective fluorescence measurements, lifetime measurements e. g. with phase modulation techniques 

and/or polarization measurements. 

generation and readout of biochemical arrays on chq3s. 

large scale surface inspections in the semiconductor industry. 

- optical recording and read-out. 

- two or multi-photon microscopy, and/or 
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- stereoscopic microscopy. 
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Plane position relative to the focal plane (^m) Figure 2 
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Figure 3 
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Figure 4 
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Mirror position relative to the focal plane (^) Figure 6 
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Fig. 7a 



Fig. 7b 
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Figure 8 



Figure 9 
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